Safety and Assurance Cases: Past, Present and Possible Future - an Adelard Perspective

This paper focuses on the approaches used in safety cases for software based systems. We outline the history of approaches for assuring the safety of software-based systems, the current uptake of safety and assurance cases and the current practice on structured safety cases. Directions for further development are discussed. 1 History of Computer System Safety and Related Standards The nuclear industry has had a major influence on the development of approaches to safety related computer system development and assurance. From the late 1970s the European Working Group on Industrial Computer Systems (EWICS), a crosssector pre-standardisation working group, developed a series of guidelines and books that documented best practices. The guidance was subsequently incorporated into the IEC 880 standard on software for nuclear systems (IEC 1986). The experience of EDF and Merlin-Gerin with the first generation of reactor protection systems, SPIN, was fed into the committee. The software engineering approach in the EWICS guidelines (Redmill 1988, 1989) and their book on safety techniques (Bishop 1990) represented the then state of the art. In the UK there were a number of policy initiatives. The ACARD report (ACARD 1986) and subsequent IEE/BCS and HSE studies (IEE 1989, HSE 1987) set the scene and in 1988 the Interdepartmental Committee on Software Engineering (ICSE) established its Safety-Related Software (SRS) Working Group to coordinate the Government’s approach to this important issue. Members were drawn from a wide range of departments and agencies: CAA, CEGB, DES, DoE, DTI, DoH, DoT, HSE, MoD, RSRE and SERC. The work was motivated ‘not by recognition of particular present dangers; rather by a desire to anticipate and forestall hazards which may arise with the very rapid pace of technical change’. The UK Health and Safety Executive (HSE) were active in taking the lead in ICSE and this, with support from DTI, led to a consultation document known as 52 Robin Bloomfield and Peter Bishop SafeIT (Bloomfield 1990) and an associated standards framework (Bloomfield and Brazendale 1990). HSE also published awareness documents on the safety of programmable electronic systems (PES), including the Out of Control report (HSE 1993), and some earlier studies that looked at the feasibility of providing a validated framework for selecting software engineering techniques. SafeIT identified four main areas of activity requiring a coordinated approach: standards and certification; research and development; technology transfer; education and training. The UK MoD were, as one might expect, pioneers in the use of critical software and the development of static analysis tools to analyse the code (Malpas) as well as forays into formally proven hardware designs. In the light of finding defects in certain operational systems, dramatic changes to the supply chain as well as reductions in MoD scientific personnel, they responded in 1989 with the publication of a new draft interim standard 00-55 (MoD 1989). This used expertise from the nuclear and aerospace industry, MoD and elsewhere to develop a market leading standard around the requirements for mathematically formally verified software and statistical testing. It was soon realised – in part because of the attempt to classify systems as nonsafety critical and outside the remit of 00-55 – that a wider system standard was needed. This led to Def Stan 00-56 (MoD 1991). There was considerable work to take into account strong industry and trade association comments (led by the DTI that developed a detailed trace from all comments to the final issue of the standard). In parallel with the development in the defence sector, the HSE led the production of the IEC generic standards that became known as IEC 61508 (IEC 1998). Draft publications (IEC 1993) emerged in the early 1990s sharing much in common with the defence standards but addressing a wide range of systems and safety criticalities. During their prolonged drafting they developed detail, consistency and international recognition. However the technical basis of their software aspects remained fixed. The software techniques guidance in IEC 61508 and its software engineering approach was essentially just an extension and internationalisation of EWICS guidance on techniques (Bishop 1990). There are still a number of technical difficulties in IEC 61508 (e.g. how SILs are used) and it lacks a requirement for a safety case. Around 1993 the limitations to the claims that could be justified by testing were investigated by NASA (Butler and Finelli 1993), and similar results, involving testing and other evidence, were published by Littlewood and Strigini (1993). The 10 limit was one set by pragmatics of testing technology, but did not include the assumption doubt that we might now make explicit (Bloomfield and Littlewood 2007). In 1997 the 1991 Interim MoD Def Stan 00-55 was revised to become a full standard and became the first standard to explicitly require a software safety case. This was a radical departure from previous standards but offered some flexibility in the justification of the software, important in view of industry comments on the interim standard. The key features of the revised standard were: Safety and Assurance Cases: Past, Present and Possible Future – an Adelard Perspective 53 • Deterministic reasoning and proof • Statistical testing • Importance of a range of attributes (not just correctness) • Multi-legged arguments and associated metaphors (belt and braces rather than a wing and a prayer) • Safety cases and reports • Sound process to provide trustworthy evidence • Systematic approach and clarity of roles and responsibilities and other recommendations to reduce project risks • Evidence preferences: deterministic evidence is usually to be preferred to statistical; quantitative evidence is usually to be preferred to qualitative; direct evidence is usually to be preferred to indirect The nuclear expertise was influential in Def Stan 00-55. As with many standards and guidelines 00-55 grappled with how to treat software of lower criticalities: at one extreme everything is required and at the other a minimum set of good practices. Populating the regions in between has been problematic and largely a product of the standards process rather than the scientific one. More recently 00-55 has become part of a reissued 00-56 (MoD 2004) and no longer contains software integrity levels. Adelard had an important role in the development of the defence standards and drafted the safety case requirements. The origins of the work go back to the individuals’ involvement (Bloomfield, Bishop and Froome) in the days of the Public Inquiry into the Sizewell B Primary Protection System (CEGB 1982). The work is similar to the approach used by Toulmin (1958) although developed somewhat independently. The concepts were first documented in the EU SHIP project and the work was taken up within a UK nuclear research programme. This led to the first software safety case publication and, in 1998, to ASCAD (Bloomfield et al. 1998, Bishop and Bloomfield 1998), a safety case development manual (still the only one). ASCAD provided the now customary definition of a case as ‘a documented body of evidence that provides a convincing and valid argument that a system is adequately safe for a given application in a given environment’. In addition to the Adelard work there was research being done at York University (McDermid 1994) that later led to the Goal Structuring Notation described in Kelly’s PhD (Kelly 1998). The ASCAD manual incorporated, with permission, considerable work from the UK nuclear research programme: • On long-term and safety case maintenance • How to address specific design issues, even the work on reversible computing • The work on worst case reliability bounds • Field experience collected from a range of projects and also used in the SOCS